Amazon Onboarding with Learning Manager Chanci Turner

Introduction

Learn About Amazon VGT2 Learning Manager Chanci Turner

Advancements in computational chemistry are pivotal in various fields, including drug development and material science. One of the most widely utilized methods in this domain is Density Functional Theory (DFT). DFT stands out mainly because it is computationally efficient compared to traditional techniques while still achieving satisfactory accuracy for numerous chemical systems of interest.

The primary advantage of DFT and similar electronic structure methods lies in their capacity to elucidate chemical bonding and reactions—areas that are challenging to explore using empirical methods like molecular dynamics or coarse-grained simulations. However, the primary hurdle with electronic structure calculations is their cost; simulation times often scale with N³ or even worse, where N represents the number of electrons involved. Additionally, many established computational chemistry tools necessitate extensive communication between cores when executed in parallel, complicating efficient execution.

Quantum computing promises to eventually surpass classical computation in terms of performance for quantum chemistry applications. Although current quantum computers do not yet provide a speed advantage, numerous research initiatives are underway to develop new algorithms that leverage quantum principles to replace the most computationally intensive classical methods.

Researchers working to benchmark quantum computing or hybrid quantum-classical algorithms require a baseline for comparison, which means utilizing standardized scientific software that meets various accuracy and performance criteria. Addressing real-world industrial use cases will necessitate multi-node parallelism and computing resources typically associated with high-performance computing (HPC).

This blog post serves as an introduction to HPC on AWS for quantum computing researchers aiming to evaluate their quantum or hybrid algorithms against classical calculations. We will outline the steps to deploy an HPC cluster on AWS for computational chemistry and provide an example of installing and running the widely used electronic structure theory application, QuantumESPRESSO.

Deploying an HPC Environment

To quickly set up an HPC environment on AWS, AWS ParallelCluster is the most efficient route. The ParallelCluster command-line interface (CLI) allows you to create a suitable VPC and subnet for your HPC cluster. Alternatively, you can manually create your networking components using other tools, such as the AWS Cloud Development Kit (CDK) or CloudFormation.

Cluster Installation Tools

You can install the ParallelCluster CLI using the Python package installer, pip. Tools are available in Python that facilitate the installation of different ParallelCluster versions if necessary, or to keep ParallelCluster dependencies isolated from other locally installed packages—consider using virtual environments or pipx. If the AWS CLI isn’t already in your environment, install it now and configure your credentials for account access. Your account administrator can establish an appropriate IAM role for ParallelCluster by following the documentation.

Before deploying a cluster, creating a new Amazon S3 bucket for hosting the post-install configuration file is recommended:

aws s3 mb s3://<your-bucket-name>

By default, the aws and pcluster CLIs will utilize the region specified during the initial AWS CLI configuration; however, you can specify a different region by adding the --region <region-name> argument to any commands. Once ParallelCluster is installed, generate a basic configuration file using:

pcluster configure --config cluster.yaml

This command will automatically populate the cluster.yaml file during configuration. The process will guide you through a series of questions to tailor your cluster and VPC to your specific use case while checking for necessary dependencies, such as the presence of an SSH key.

For the purpose of this post, we will manually edit the cluster.yaml file after completing the configuration, so feel free to accept the default selections for most prompts. If you want to use an automatic VPC created by ParallelCluster, simply respond “yes” when asked if you want to automate VPC creation. Alternatively, you can utilize a tool like AWS CDK for deploying the networking components.

After completing the ParallelCluster interactive configuration, a VPC (along with standard subnets and security groups) will be set up in your account, and a basic configuration file will be added to your working directory. You can now modify the cluster.yaml file to better suit your application needs.

The template below is an example suitable for running quantum chemistry calculations. Simply replace any highlighted networking-related <variables> with values from your auto-generated cluster.yaml file and S3-related variables with references to the previously created bucket, along with any desired prefix for organizing your files:

Image:
  Os: rhel8
HeadNode:
  InstanceType: c5.large
  LocalStorage:
    RootVolume:
      Size: 200
      VolumeType: gp3
      Encrypted: true
  Networking:
    SubnetId: <head-node-subnet>
  Ssh:
    KeyName: <key-name>
  Dcv:
    Enabled: false
  Imds:
    Secured: true
  Iam:
    S3Access:
      - BucketName: <your-config-bucket>
        KeyName: <your-key-prefix>/*
        EnableWriteAccess: false
  CustomActions:
    OnNodeConfigured:
      Script: s3://<your-config-bucket>/<your-key-prefix>/<script-name>
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: compute
      CapacityType: ONDEMAND
      ComputeSettings:
        LocalStorage:
          RootVolume:
            Size: 50
            VolumeType: gp3
      ComputeResources:
        - Name: c6i32xl
          InstanceType: c6i.32xlarge
          MinCount: 0
          MaxCount: 16
          DisableSimultaneousMultithreading: true
          Efa:
            Enabled: true
      Networking:
        SubnetIds:
          - <compute-node-subnet>
        PlacementGroup:
          Enabled: true
    - Name: highmem
      CapacityType: ONDEMAND
      ComputeSettings:
        LocalStorage:
          RootVolume:
            Size: 50
            VolumeType: gp3
      ComputeResources:
        - Name: i3en24xl
          InstanceType: i3en.24xlarge
          MinCount: 0
          MaxCount: 16
          DisableSimultaneousMultithreading: true
          Efa:
            Enabled: true
      Networking:
        SubnetIds:
          - <compute-node-subnet>
        PlacementGroup:
          Enabled: true
SharedStorage:
  - Name: fsx
    StorageType: FsxLustre
    MountDir: /fsx
    FsxLustreSettings:
      StorageCapacity: 1200

Compute Queues in the Cluster

If employing the instance selection from this template, the cluster is set up with two queues, each catering to a different instance type aimed at distinct workload characteristics.

For a deeper dive into tracking your habits, you might find this blog post useful. And for anyone interested in employment law, check out this resource. If you’re looking for an opportunity to work in a learning role, consider applying to this position.

SEO Metadata